\chapter{Experimentation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Sub-Table of Content
%  1- Experimental Setup
%  2- Experimental Procedure
%  2- Results
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%  Experimental Setup

\section{Experimental Setup}

\subsection{Test Configuration}

The results have been gathered using the following configuration:

\begin{itemize}
  \item \textbf{CPU:} Intel(R) Pentium(R) M processor 1.80GHz
  \item \textbf{RAM:} 1GB
  \item \textbf{OS:} GNU/Linux Gentoo
  \begin{itemize}
    \item \textbf{CFLAGS:} -O2 -march=pentium-m -pipe -fomit-frame-pointer
  \end{itemize}
  \item \textbf{VM:} VirtualBox-2.0.4
  \begin{itemize}
    \item \textbf{Guest OS:} Linux Gentoo
    \item \textbf{Base Memory:} 256MB
    \item \textbf{Video Memory:} 16MB
    \item \textbf{Network:} PCnet-FAST III (host interface, tap0)
  \end{itemize}
\end{itemize}

There are two different batteries of tests to demonstrate the capabilities
of the \textbf{Lemona} system:

\begin{itemize}
  \item Performance \& Availability Benchmarks
  \item Functional Tests
\end{itemize}


All tests have been carried using a statically compiled lemona module
using different configurations:

\begin{itemize}
  \item lemona + net

  The network module is enabled, but there is no server to send data to.

  \item lemona + net + basket

  The network module is enabled and the \emph{basket} server is
  accepting data.

  \item lemona + relay

  The \emph{kernel}'s relay facility is enabled, but no program is
  reading the output.

  \item lemona + relay + cat

  The \emph{kernel}'s relay facility is enabled and the output is read
  by cat to \emph{/dev/null}.
\end{itemize}


\subsection{Performance \& Availability Benchmarks}

These benchmarks aim to demonstrate the usability of \textbf{Lemona} in a
test environment, by proving that its performance impact does not render
the system unavailable.

We want to check the performance impact of the \textbf{Lemona} system for
"standard" system usage, based on tests presented by \textbf{Forensix} \cite{goel:FORENSIX}:

\begin{itemize}
  \item benchmarking the build of a Linux \emph{kernel} from scratch
  \item benchmarking the throughput of an \textbf{Apache Web Server}.
\end{itemize}

Each benchmark set is carried using 12 different configurations:

\begin{itemize}
  \item \textbf{Lemona} has neither been built, nor loaded
  \item \textbf{Lemona} has been built as a module, but the module is not loaded
  \item \textbf{Lemona} has been built as a module and the module has been loaded:
  \begin{itemize}
    \item Relay reporting is enabled / Socket reporting is disabled
    \item Relay reporting is disabled / Socket reporting is enabled with encryption
    \item Relay reporting is disabled / Socket reporting is enabled without encryption
    \item Relay and Socket reporting are enabled (with encryption)
    \item Relay and Socket reporting are enabled (without encryption)
  \end{itemize}
  \item \textbf{Lemona} has been built statically:
  \begin{itemize}
    \item Relay reporting is enabled / Socket reporting is disabled
    \item Relay reporting is disabled / Socket reporting is enabled with encryption
    \item Relay reporting is disabled / Socket reporting is enabled without encryption
    \item Relay and Socket reporting are enabled (with encryption)
    \item Relay and Socket reporting are enabled (without encryption)
  \end{itemize}
\end{itemize}

Recorded information for the performance benchmarks include:

\begin{itemize}
  \item timestamps for:
  \begin{itemize}
    \item benchmark start
    \item benchmark end
    \item each of the following readings
  \end{itemize}
  \item regular and peak readings for:
  \begin{itemize}
    \item I/O throughput levels (in bits per seconds)
    \item CPU consumption levels (in percentage of available processing power)
    \item Memory consumption levels (in bytes and percentage of available memory)
    \item database inputs (in records per minutes)
    \item average and peak log file sizes
    \item average and total number of database entries
  \end{itemize}
\end{itemize}

\subsection{Functional Tests}

These tests aim to demonstrate the functional feasibility of \textbf{Lemona}
within a real-life environment. Using real case scenarios (yet to be
determined), the application will be tested to assert:

\begin{itemize}
  \item \textbf{Lemona}'s capacity at collecting relevant information

  Information relative to security breaches or system failures should
  be recorded on the host targeted and stored in the database to be
  identified easily for later forensics research.

  \item \textbf{Lemona}'s capacity at retrieving relevant information

  The information collected during the first phase of the test should
  be easily and quickly accessed using appropriate queries.
\end{itemize}

Recorded information for the functional tests include:

\begin{itemize}
  \item timestamps for:
  \begin{itemize}
    \item benchmark start
    \item benchmark end
    \item each of the following actions upon start and completion
  \end{itemize}
  \item readings for:
  \begin{itemize}
    \item returned noise and false positives (absolute and relative reading)
    \item lookup time (in seconds)
  \end{itemize}
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%  Experimental Procedure

\section{Experimental Procedure}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%  Results

\section{Results}

\subsection{Notes on the \textbf{Lemona} Results}

As of this writing, results are still being collected. We are just
providing below some insights on our expectations and our
checkpoints, as well as the current results we managed to collect.

Currently, we have not collected the results for the functional tests
outlined above, as \textbf{Lemona} is not feature complete from the computer
forensics point of view, and we did not perform tests with an active
encryption layer, as it is not yet part of our proof of concept.

Regarding the benchmark comparison with \textbf{Forensix}, we would
like to emphasize that they might not be easy to compare with a 1-to-1
ratio, for the following reasons:

\begin{itemize}
  \item \textbf{Coverage}

  \textbf{Forensix} does not record every single operation, and only
  focuses on specific processes, whereas \textbf{Lemona} aims at
  monitoring the complete system activity. On the other hand,
  \textbf{Lemona} does not (yet) monitor all the \emph{system calls};
  but the ones implemented are monitored for all processes.

  \item \textbf{Hardware and Software Configurations}

  We do not have at our disposal the hardware resources to perform
  tests matching these of \textbf{Forensix}. Therefore, we use
  different hardware and system configurations, and our tests were run
  within a \emph{virtual machine}.

  \item \textbf{Tests Replicability}

  \textbf{Lemona} and \textbf{Forensix} are compared based on similar
  actions, but with different inputs. For instance, they used
  different kernel sources for the performance test, which means that
  the kernel compilation time might not be as relevant.

\end{itemize}

However, we believe these benchmarks provide a valuable insight and
indication of \textbf{Lemona}'s usability on a real system, and that
the transparence we provide with this comparison \textbf{Forensix}
might allow readers to determine if our solution matches their needs.

\subsection{Coverage}

We intend to cover close to 100\% of a system's activity, in that we
actually trace all existing \emph{system calls} and monitor memory
mapped areas' read and write accesses. We should therefore be able to
monitor all actions executed by all (human and machine) users logged
onto a system as the system's core activity. We expect to be able to
collect enough data to virtually reconstruct a complete system from a
given checkpoint.

Our proof of concept achieves a technical validation of this
objective, by monitoring twenty-two (22) \emph{system calls}, among
which some of the most intensive ones, performing I/O operations.

\subsection{Performance}

\subsubsection{CPU: Linux Kernel 2.6.26.3 Compilation}

The kernel has been compiled using the configuration file found in
\emph{arch/x86/configs/i386\_defconfig\_} that comes along with its
sources.

Between each test:

\begin{itemize}
  \item the virtual machine has been reset
  \item the 'old' Linux directory has been deleted and copied over again.
\end{itemize}

For each test \emph{make menuconfig} has been executed before
launching the compilation. No modification has been made to the
default configuration, we simply have him save the \emph{.config}
file.

We use the \emph{time make} command to produce the following
measurements.

\begin{table}
  \renewcommand{\arraystretch}{1.3}
  \caption{Performance Results}
  \label{table:performance}
  \centering
  \begin{tabular}{ | l | l | l | l | l | }
    \hline
    \textbf{Test}                     & \textbf{User Time} & \textbf{System Time} & \textbf{Real Time} & \textbf{CPU}  \\ \hline
    \textbf{w/o lemona}               & 283s               & 226s                 & 519s              & 98\%           \\ \hline
    \textbf{w/ lemona + net}          & 242s               & 256s                 & 531s  (+02.31\%)  & 93\%           \\ \hline
    \textbf{w/ lemona + net + basket} & 269s               & 334s                 & 670s  (+29.09\%)  & 90\%           \\ \hline
    \textbf{w/ lemona + relay}        & 300s               & 292s                 & 604s  (+16.37\%)  & 98\%           \\ \hline
    \textbf{w/ lemona + relay + cat}  & 247s               & 256s                 & 1028s (+98.07\%)  & 49\%           \\ \hline
    \hline
  \end{tabular}
\end{table}

As we predicted, we notice a significant impact on the system's
performance when \textbf{Lemona} is enabled. However, and even though
\textbf{Lemona} performs worse than \textbf{Forensix} (See Appendix
A), our proof of concept remains usable inspite of the overhead.

\subsubsection{Network: Apache 2 Benchmark}

An \textbf{Apache 2} server has been installed, default configuration has
been kept, and a 169 bytes \emph{index.html} file created.

The benchmark has been done using the 'ab' tool from the
\textbf{Apache Foundation} as follows:

\begin{verbatim}
ab -n 10000 -c 10 http://10.0.42.2/
\end{verbatim}

\begin{table}
  \renewcommand{\arraystretch}{1.3}
  \caption{Network Results}
  \label{table:network}
  \centering
  \begin{tabular}{ | l | l | l | l | }
    \hline
    \textbf{Test}                     & \textbf{Total Time} & \textbf{Requests/s} & \textbf{Real Time}   \\ \hline
    \textbf{w/o lemona}               & 5.767s              & 1733.97             & 687.49Kb             \\ \hline
    \textbf{w/ lemona + net}          & 5.977s (+03.64\%)   & 1673.00 (-03.51\%)  & 663.32Kb (-03.51\%)  \\ \hline
    \textbf{w/ lemona + net + basket} & 6.912s (+19.85\%)   & 1446.73 (-16.56\%)  & 573.61Kb (-16.56\%)  \\ \hline
    \textbf{w/ lemona + relay}        & 6.091s (+05.61\%)   & 1641.63 (-05.32\%)  & 650.88Kb (-05.32\%)  \\ \hline
    \textbf{w/ lemona + relay + cat}  & 6.700s (+16.17\%)   & 1492.50 (-13.92\%)  & 591.75Kb (-13.92\%)  \\ \hline
    \hline
  \end{tabular}
\end{table}

As for the CPU consumption, the network resources take a significant
hit if \textbf{Lemona} is configured to transmit its traces over the
network. Considering \textbf{Lemona} will be transmitting all the
traces over a network interface, it will be under considerable
load. However, we might implement buffering techniques to improve
this, and we think the solution should provide satisfactory results to
be used on a \emph{LAN}. Benchmarks will show if this is a viable
solution for systems residing on more distant networks, such as
\emph{WAN}/\emph{VPNs}.

\subsubsection{Memory}

As of today, the tracing structures' memory footprint is less than 30
bytes, if they do not require additional parameters. Regarding the
memory consumption on the storage point's side, which we expect to
have a high rate, we are not able at the moment to provide valuable
benchmarks, for various reasons, which are explained below.

\begin{itemize}
  \item \textbf{Lemona}'s internal storage structures are morphing because:
  \begin{itemize}
    \item We are still in the process of modifying our API;
    \item They depend on which system call is being traced.
  \end{itemize}
  \item The rate at which the traces will be generated is really variable because:
  \begin{itemize}
    \item It depends on the kind of load the monitored host is being put under;
    \item It depends on the kind of activity the monitored host is performing (watching a movie or doing a full-text search will not have the same system call throughputs)
  \end{itemize}
\end{itemize}
